In view of the problem that it is difficult to accurately recognize the type of vehicle due to scale change and deformation under multiple angles, a fine-grained vehicle recognition model based on Multi-Scale Bilinear Convolutional Neural Network (MS-B-CNN) was proposed. Firstly, B-CNN was improved and then MS-B-CNN was proposed to realize the multi-scale fusion of the features of different convolutional layers to improve feature expression ability. In addition, a joint learning strategy was adopted based on center loss and Softmax loss. On the basis of Softmax loss, a category center was maintained for each category of the training set in the feature space. When new samples were added in the training process, the classification center distances of samples were constrained to improve the ability of vehicle recognition in multi-angle situations. Experimental results show that the proposed vehicle recognition model achieved 93.63% accuracy on CompCars dataset, verifying the accuracy and robustness of the model under multiple angles.